Image processing apparatus, image processing apparatus control method,  image pickup apparatus, and image pickup apparatus control method

ABSTRACT

An image processing apparatus including an acquiring unit configured to acquire an image signal of a plurality of images obtained by picking up an object image that is formed by a photographing optical system; a determining unit configured to determine a defocus amount of the object image by using the image signal of the plurality of images; and a conversion unit configured to perform, on the defocus amount, gradation conversion that has at least one conversion characteristic out of a plurality of different conversion characteristics, and to output information that is based on the defocus amount converted by the gradation conversion.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing apparatus control method, an image pickup apparatus, and animage pickup apparatus control method.

2. Description of the Related Art

There has been known a type of image pickup apparatus that uses a pupildivision-type phase difference detecting method or a plurality of imagesdifferent from one another in blurring to detect a defocus amount (thedifference between a light receiving plane and an image forming plane ofa lens).

A method of calculating the defocus amount from the image deviationamount (parallax amount) of an image is disclosed in Japanese PatentApplication Laid-Open No. 2008-15754. An image is divided into minuteblocks, a correlation value is calculated by deviating data relative toa pair of pieces of pixel data in one of the minute blocks, and adeviation amount at which the correlation is highest is obtained as theparallax amount. From the calculated deviation amount and a conversionefficient, which is determined based on an image pickup element's pixelpitch and a lens that are used, the defocus amount is calculated withrespect to an expected image forming plane of an object image plane.

A method of calculating the defocus amount by Depth from Defocus (DFD)is disclosed in Japanese Patent Application Laid-Open No. 2013-253964.In DFD, a photographing parameter of an image pickup optical system iscontrolled so as to obtain a plurality of images different from oneanother in blurring, and the amount of correlation in blurring between ameasurement target pixel and its surrounding pixels in the plurality ofobtained images is calculated to calculate the defocus amount. Thedefocus amount calculated by these methods is a distance on the imageplane, and object distance information can therefore be calculatedfurther by converting the image plane distance into an object planedistance with the use of the lens equation.

Technologies of applying the defocus amount and the object distance thatare calculated by these methods to various types of image processinghave been disclosed as well. In Japanese Patent Application Laid-OpenNo. 2010-45457, face brightness correction includes performingprocessing in which a correction gain is changed in relation to thedefocus amount so that the face in focus has an appropriate brightness.Japanese Patent Application Laid-Open No. 2013-243529 includesperforming processing in which an image pickup apparatus uses the objectdistance to pick up an image of an object when the object is within aprescribed distance from the image pickup apparatus.

SUMMARY OF THE INVENTION

According to an aspect of an embodiment, an image processing apparatusincluding an acquiring unit configured to acquire an image signal of aplurality of images obtained by picking up an object image that isformed by a photographing optical system; a determining unit configuredto determine a defocus amount of the object image by using the imagesignal of the plurality of images; and a conversion unit configured toperform, on the defocus amount, gradation conversion that has at leastone conversion characteristic out of a plurality of different conversioncharacteristics, and to output information that is based on the defocusamount converted by the gradation conversion.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for illustrating the system configuration of animage pickup apparatus to which an image processing apparatus accordingto a first embodiment of the present invention is applied.

FIG. 2 is a diagram for schematically illustrating a pixel arrangementof an image pickup element in the image pickup apparatus according tothe first embodiment of the present invention.

FIG. 3 is a block diagram of an image processing unit of the imagepickup apparatus according to the first embodiment of the presentinvention.

FIGS. 4A, 4B, and 4C are graphs for showing image deviation amountdetection that uses an image having parallax.

FIGS. 5A, 5B, 5C, and 5D are graphs for showing examples of gradationconversion characteristics in the image pickup apparatus according tothe first embodiment of the present invention.

FIG. 6 is a diagram for illustrating the relation between an F-value, animage deviation amount, and a defocus amount.

FIG. 7 is a block diagram of an image processing unit of an imageprocessing apparatus according to a second embodiment of the presentinvention.

FIG. 8 is a graph for showing another example of the gradationconversion characteristics in the image processing apparatus accordingto the first embodiment of the present invention.

FIG. 9 is a diagram for illustrating the relation between an imagepickup plane distance and an object plane distance.

FIG. 10 is a graph for showing an example of gradation conversioncharacteristics in the image processing apparatus according to thesecond embodiment of the present invention.

FIG. 11 is a diagram for illustrating another example of the pixelarrangement of the image pickup element.

FIG. 12 is a block diagram of an image processing unit of an imageprocessing apparatus according to a third embodiment of the presentinvention.

FIGS. 13A and 13B are block diagrams of an image processing unit of animage processing apparatus according to a fourth embodiment of thepresent invention.

FIGS. 14A, 14B, 14C, 14D, and 14E are diagrams for illustrating imagegeneration in the image processing apparatus according to the fourthembodiment of the present invention.

FIGS. 15A, 15B, 15C, and 15D are graphs for showing the relation betweenan object distance and a distance map in the image processing apparatusaccording to the fourth embodiment of the present invention.

FIG. 16 is a diagram for illustrating the relation between the objectdistance and a background magnification ratio in the image processingapparatus according to the fourth embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

When the calculated defocus amount and object distance are output to theoutside of an image pickup apparatus, or output to another module orapplication within the image pickup apparatus, the distance informationneeds to be converted before being output, depending on where thedistance information is output to. For instance, some need a distanceresolution near the in-focus region as in Japanese Patent ApplicationLaid-Open No. 2010-45457, and others need the distance measurement rangeas in Japanese Patent Application Laid-Open No. 2013-243529. Thedistance information also needs to be kept under the limit word lengthof a transmission path that connects to the output destination of thedistance information.

Exemplary embodiments of the present invention are described in detailbelow with reference to the drawings. The following embodiments describeexamples of applying an image processing apparatus of the presentinvention to a digital camera, which is an example of an image pickupapparatus.

First Embodiment

FIG. 1 is a block diagram for illustrating the function configuration ofa digital camera as an example of an image pickup apparatus to which animage processing apparatus according to a first embodiment of thepresent invention is applied. In FIG. 1, a control unit 101 is, forexample, a CPU, and controls the operation of blocks that are includedin a digital camera 100 by reading the blocks' respective operationprograms out of a ROM 102, and deploying the operation programs on a RAM103 to perform the operation programs. The ROM 102 is a rewritablenon-volatile memory, and stores, in addition to the operation programsof the blocks included in the digital camera 100, parameters and othertypes of information necessary for the operation of the blocks. The RAM103 is a rewritable volatile memory, and is used as a temporary storagearea for data that is output through the operation of the blocksincluded in the digital camera 100.

A photographing optical system 104 forms an image of an object in animage pickup unit 105. The image pickup unit 105 includes an imagepickup element such as a CCD or a CMOS sensor, and outputs, to an A/Dconversion unit 106, an analog image signal, which is obtained throughphotoelectric conversion of an optical image formed in the image pickupelement by the photographing optical system 104. The A/D conversion unit106 performs A/D conversion processing on the input analog image signal,and outputs the resultant digital image data to the RAM 103, where thedigital image data is stored.

An image processing unit 107 performs various image processing such aswhite balance adjustment, color interpolation, reduction/enlargement,and filtering on image data stored in the RAM 103.

A recording medium 108 is a detachable memory card or the like. Imagesstored in the RAM 103, including an image that has been processed by theimage processing unit 107 and an image that has been subjected to A/Dconversion by the A/D conversion unit 106, are recorded on the recordingmedium 108 as recorded images. The image processing unit 107 and therecording medium 108 also handle the generation and recording of adefocus amount and an object distance, which are described later.

FIG. 2 is a diagram for schematically illustrating the pixel arrangementconfiguration of the image pickup unit 105 that is included in thedigital camera 100 according to this embodiment. The pixel arrangementconfiguration of FIG. 2 is the same as that of the image pickup elementdisclosed in Japanese Patent Application Laid-Open No. 2008-15754. InFIG. 2, each unit pixel 202 includes a micro lens 201 and a pair ofphotoelectric converters 203 and 204. The photoelectric converters(divided pixels) 203 and 204 receive luminous fluxes via the micro lens(pupil dividing unit) 201 from different pupil regions of an exit pupilof the photographing optical system 104. In the image pickup unit 105 ofFIG. 1 where the unit pixels 202 are arranged two-dimensionally andorderly, an image signal of an image is generated from a luminous fluxthat has passed through a pupil division region associated with, forexample, the divided pixel 203 by collecting the photoelectricconversion output of the divided pixel 203 from each unit pixel 202.Similarly, an image signal of an image is generated from a luminous fluxthat has passed through another pupil division region, which isassociated with the other divided pixel 204, by collecting thephotoelectric conversion output of the divided pixel 204. The imagepickup unit 105 can acquire a pair of image signals having parallax inthis manner. In this embodiment, an image picked up by the photoelectricconverter 203 is referred to as image A and an image picked up by thephotoelectric converter 204 is referred to as image B.

FIG. 3 is a diagram for illustrating an image generation unit, a defocusamount calculation unit, and a gradation conversion unit in the imageprocessing unit 107 of the digital camera to which the image processingapparatus according to this embodiment is applied.

The image generation unit 305 adds together a plurality of object imagesformed from luminous fluxes that arrive from different regions of thepupil of the photographing optical system 104 (the image A and the imageB in this embodiment), and generates an image corresponding to a singleobject image that is generated from luminous fluxes arriving from allregions of the pupil of the photographing optical system 104. The imageA that is denoted by 306 and the image B that is denoted by 307 areinput to the image generation unit 305, and the image generation unit305 outputs an image 308. The image 308 generated by the imagegeneration unit 305 receives further image processing such as whitebalance adjustment, color interpolation, reduction/enlargement, andfiltering, and the processed image is recorded on the recording medium108 of FIG. 1 as a recorded image.

The defocus amount calculation unit, which is denoted by 301, isconfigured to calculate a defocus amount at a target pixel position. Theimage A that is denoted by 302 and the image B that is denoted by 303are input to the defocus amount calculation unit 301, and the defocusamount calculation unit 301 outputs a defocus amount 304. The defocusamount calculation unit 301 calculates the defocus amount by calculatinga correlation function of correlation between the image A and the imageB, or by other methods.

Processing of calculating the correlation function is described. Theprocessing of calculating the defocus amount from the correlationfunction can use a method disclosed in Japanese Patent ApplicationLaid-Open No. 2008-15754. Concrete processing of the calculation isdescribed with reference to FIGS. 4A to 4C. In each of FIGS. 4A to 4C,the axis of abscissa represents the image deviation amount and the axisof ordinate represents the amount of correlation between images that isexpressed by Expression (1). FIG. 4A is an example in which the degreeof correlation is high. FIGS. 4B and 4C are examples in which the degreeof correlation is low.

According to Japanese Patent Application Laid-Open No. 2008-15754, apair of pieces of pixel data in one minute block are expressed in ageneralized form as (E(1) to E(m)) and (F(1) to F(m)) (m represents thenumber of data pieces). The data series (F(1) to F(m)) is deviatedrelative to the data series (E(1) to E(m)) to calculate, by Expression(1), a correlation amount C(k) between the two data series at an imagedeviation amount k.

C(k)=Σ|E(n)−F(n+k)|  (1)

In Expression (1), Σ operation is calculated for n. Values that n andn+k can take in this operation are limited to a range of 1 to m. Theimage deviation amount k is an integer, and indicates the amount ofrelative shift between images.

The calculation result of Expression (1) is as shown in FIG. 4A, and thecorrelation amount C(k) is smallest at a shift amount where correlationbetween a pair of data series is high (the degree of correlation ishigher when the correlation amount is smaller). Subsequently,three-point interpolation by Expressions (2) to (5) is used to obtain ashift amount x that gives a minimum value C(x) for successivecorrelation amounts.

x=kj+D/SLOP  (2)

C(x)=C(kj)−|D|  (3)

D={C(kj−1)−C(kj+1)}/2  (4)

SLOP=MAX{C(kj+1)−C(kj), C(kj−1)−C(kj)}  (5)

The shift amount x obtained by Expression (2) can be used to obtain adefocus amount DEF with respect to an expected image forming plane of anobject image plane by Expression (6).

DEF=KX·PY·x  (6)

In Expression (6), PY represents the pixel pitch of the image pickupelement (the distance between pixels that constitute the image pickupelement), and KX represents a conversion coefficient that is determinedbased on the magnitude of the divergence angle at the barycenter ofluminous fluxes that pass through a pair of distance measurement pupils.The unit of the conversion coefficient is mm/pix. The magnitude of thedivergence angle at the barycenter of luminous fluxes that pass througha pair of distance measurement pupils varies depending on the size ofthe diaphragm opening (F-value) of the lens, and is therefore determinedbased on lens information. The details of this relation are describedwith reference to FIG. 6. FIG. 6 is a diagram for illustrating therelation between the defocus amount with respect to the expected imageforming plane, the F-value, and the image deviation amount. Illustratedin FIG. 6 are an object plane 601 of an image pickup object, a lens 602,an expected image forming plane 603 of the image pickup element, and animage plane 604, which is located at a position of defocus by a defocusamount 605 from the expected image forming plane 603. Image deviationamounts 607 and 608, which are in relation to the F-value, are alsoillustrated in FIG. 6. Specifically, the image deviation amount 607 isobserved when the F-value is small (the diaphragm opening is wide), andthe image deviation amount 608 is observed when the F-value is large(the diaphragm opening is narrow). It is understood from FIG. 6 that theimage deviation amount with respect to the image plane at a defocusedposition is larger when the diaphragm opening is wider and smaller whenthe diaphragm opening is narrower. In other words, the image deviationamount varies depending on the F-value even at the same defocus amount.This is why the image deviation amount is multiplied by KX, which is again that is in relation to the F-value, when the defocus amount iscalculated from the image deviation amount.

Whether or not the calculated defocus amount DEF is reliable isdetermined as follows. When the degree of correlation between a pair ofdata series is low, the minimum value C(x) of the interpolatedcorrelation amount is large as shown in FIG. 4B. The reliability istherefore determined as low when C(x) is equal to or more than aprescribed value. Alternatively, the reliability is determined as lowwhen a value that is obtained by normalizing C(x) with the contrast ofdata is equal to or more than a prescribed value. The normalization isaccomplished by dividing C(x) by an SLOP value that is in proportion tothe contrast. In the case where an SLOP value that is in proportion tothe contrast is equal to or less than a prescribed value, it isdetermined that the object is low in contrast and the reliability of thecalculated defocus amount DEF is low.

When the degree of correlation between a pair of data series is low andthe correlation amount C(k) does not drop in a prescribed shift range offrom k min to k max as shown in FIG. 4C, the minimum value C(x) cannotbe obtained. In such cases, it is determined that the image deviationamount cannot be detected. In the case where the image deviation amountcan be detected, the defocus amount is calculated by Expression (6).

The sum of absolute differences (SAD) between pixels is used for thecorrelation function in Japanese Patent Application Laid-Open No.2008-15754. However, this embodiment is not limited thereto. Forexample, the sum of squared differences (SSD) or normalized crosscorrelation (NCC) between pixels may be used. The NCC value is largerwhen the degree of correlation is higher, whereas the SAD value and theSSD value are smaller when the degree of correlation is higher. Otherthan SAD and NCC, any correlation operation expression that calculatesthe degree of match between a pair of object images can be used.

As illustrated in FIG. 3, the defocus amount 304 output from the defocusamount calculation unit 301 is input to the gradation conversion unit309. When the defocus amount 304 is output, the gradation conversionunit 309 performs data conversion suitable for where the defocus amount304 is output to, or for a standard that is used. The gradationconversion unit 309 outputs a defocus amount 310 on which gradationconversion has been performed (the output is referred to as “defocus mapoutput”). The gradation conversion is described with reference to FIGS.5A to 5D on the premise that an interface to the defocus amount outputdestination uses 8-bit (0 to 255) gradation in this embodiment. FIGS. 5Ato 5D are graphs for showing the characteristics of conversion from thedefocus amount to the defocus map in the gradation conversion unit 309.The axis of abscissa represents the defocus amount and the axis ofordinate represents the defocus map in each of the graphs. An object forwhich a defocus amount of 0 [mm] is detected is a focused object. Apositive defocus amount indicates an object that is in front of thefocused object, and the distance from the focused object toward the nearside grows when the defocus amount increases in the positive direction.A negative defocus amount indicates an object that is behind the focusedobject, and the distance from the focused object toward the far sidegrows when the defocus amount increases in the negative direction. Inconversion characteristics 501 of FIG. 5A, linear gradation conversionis performed on the defocus amount 304. In conversion characteristics502 of FIG. 5B, on the other hand, non-linear gradation conversion isperformed on the defocus amount 304 so that more bits are allocated to adefocus amount around 0 [mm], which is near an in-focus region, than inother regions. In the case where the output destination of the defocusamount is an application that requires a wide distance measurement rangeas in Japanese Patent Application Laid-Open No. 2013-243529, lineargradation conversion having the conversion characteristics 501 or thelike is performed to allocate bits equally to any defocus amount. In thecase where the output destination of the defocus amount is anapplication that requires distance resolution near the in-focus regionas in Japanese Patent Application Laid-Open No. 2010-45457, on the otherhand, non-linear gradation conversion having the conversioncharacteristics 502 or the like is performed to allocate more bits to adefocus amount near the in-focus region than in other regions. Thenon-linear characteristics of the conversion characteristics 502 ismerely an example, and the gradation conversion may have non-linearcharacteristics that are asymmetrical with respect to a defocus amountof 0 [mm], such as conversion characteristics 503 of FIG. 5C (more bitsare allocated to an object that is behind the focused object than to anobject that is in front of the focused object). Conversioncharacteristics 504 of FIG. 5D or the like may also be used in the caseof an application that needs the defocus amount of a specific object(for example, an object specified by a user).

These candidates for gradation conversion characteristics to be employedby the gradation conversion unit 309 may be stored in advance in the ROM102 as table data that holds an output destination or a standard andgradation conversion characteristics in association with them.Alternatively, the gradation conversion characteristics candidates maybe received from an external apparatus that is the output destination ofthe defocus amount, such as a PC, a printer, or a mobile device, whilethe digital camera 100 is connected to the external apparatus, or from aserver while the digital camera 100 is connected to the server. In thecase where a standard is selected, gradation conversion characteristicscandidates that are associated with different standards may be stored inthe ROM 102 in advance. The gradation conversion characteristics thatare employed by the gradation conversion unit 309 may be generated eachtime gradation conversion is performed, based on information aboutobject detection, about the output destination of the defocus amount, orabout the standard that is used. An output destination or a standard towhich the gradation conversion characteristics are to be adapted isselected from a plurality of output destination candidates stored inadvance in the ROM 102, or is determined based on information about theexternal apparatus given above, or about a server connected to thedigital camera 100, such as an output bit count or a standard that isassociated with the external apparatus or the server, by obtaining theinformation from the external apparatus or the server while the digitalcamera 100 is connected to the external apparatus or the server.

As described in the description of Expression (6), the image deviationamount is multiplied by the gain KX, which is in relation to theF-value, when the defocus amount is calculated from the image deviationamount. The gradation characteristics can therefore be varied dependingon the F-value. This adjustment is described with reference to FIG. 8.FIG. 8 is a graph for showing the characteristics of conversion from thedefocus amount to the defocus map as FIGS. 5A to 5D are. The conversioncharacteristics of FIG. 8 are linear gradation conversioncharacteristics that are adjusted based on the F-value. Shown in FIG. 8are gradation characteristics 801 at an F-value of 1.8, gradationcharacteristics 802 at an F-value of 3.5, and gradation characteristics803 at an F-value of 5.6. Image deviation that is generated at the samedefocus amount is larger in amount when the F-value is smaller, and asmaller F-value therefore means a wider distance measurement range.Accordingly, the gradation characteristics 801 (F=1.8) are close tolinear characteristics and, when the conversion characteristics areclose to the gradation characteristics 802 (F=3.5) and the gradationcharacteristics 803 (F=5.6), fewer bits are allocated to a largerdefocus amount. The F-value is not the only cause of fluctuations indistance measurement range, and the base length at the time images areobtained is another factor for the fluctuations. A longer base lengthmeans greater parallax and a wider distance measurement range, and thegradation characteristics can therefore be varied depending on the baselength. In the case where the distance measurement range or the distancemeasurement resolution varies depending on a photographing conditionsuch as the ISO sensitivity of the image pickup element, gradationcharacteristics associated with the photographing condition may be used.

The image pickup unit (sensor) 105 of the image pickup apparatusconfigured to acquire image signals having parallax is not limited tothe pixel arrangement of FIG. 2, and may have a pixel arrangementstructure illustrated in FIG. 11. In FIG. 11, each pixel 1106 includes amicro lens 1105 and two pairs of photoelectric converters, 1101 and1104, and 1102 and 1103. The pixels 1106 are arranged two-dimensionallyand orderly in the image pickup unit 105. The photoelectric converter1101 picks up an image A. The photoelectric converter 1102 picks up animage B. The photoelectric converter 1103 picks up an image C. Thephotoelectric converter 1104 picks up an image D. An image and a defocusmap are generated from the image A, the image B, the image C, and theimage D as in FIG. 3.

While an example of using a pupil division image pickup element is usedfor the defocus amount calculation has been described here, the defocusamount calculation is not limited thereto. The defocus amount may becalculated with the use of images that are acquired from a compound-eyecamera in which lenses and an image pickup element are arranged instereo. DFD described in Japanese Patent Application Laid-Open No.2013-253964 and related art may be used instead of calculating thedefocus amount from the image deviation amount of an image. In DFD, thedefocus amount is calculated from a blurring evaluation value, which isbased on the powers of a plurality of images different from one anotherin blurring.

According to this embodiment, it is therefore possible to provide theimage processing apparatus and the image pickup apparatus capable of,when a defocus amount is output, outputting a defocus amount that hasgradation characteristics suited to the output destination of thedefocus amount. According to this embodiment, the defocus amount outputby the image processing apparatus and the image pickup apparatus cantherefore be used favorably where the defocus amount is output to.

Second Embodiment

A digital camera as an image pickup apparatus to which an imageprocessing apparatus according to a second embodiment of the presentinvention is applied is described below. The configuration and the imagepickup element of the digital camera as an example of the image pickupapparatus in this embodiment are the same as those described in thefirst embodiment with reference to FIGS. 1 and 2, and a descriptionthereof is omitted here unless it is necessary. The second embodimentdiffers from the first embodiment in that output information of thegradation conversion unit is the object distance (object planedistance).

FIG. 7 is a diagram for illustrating an image generation unit, a defocusamount calculation unit, an object distance calculation unit, and agradation conversion unit in the image processing unit 107 of thedigital camera 100 to which the image processing apparatus according tothis embodiment is applied. The image generation unit 705 adds togethera plurality of object images formed from luminous fluxes that arrivefrom different regions of the pupil of the photographing optical system,and generates a single object image that is generated from luminousfluxes arriving from all regions of the pupil of the photographingoptical system. An image A that is denoted by 706 and an image B that isdenoted by 707 are input to the image generation unit 705, and the imagegeneration unit 705 outputs an image 708. The image 708 generated by theimage generation unit 705 receives further processing such as whitebalance adjustment, color interpolation, reduction/enlargement, andfiltering. The processed image is recorded on the recording medium 108of FIG. 1 as a recorded image. The defocus amount calculation unit 701is configured to calculate a defocus amount at a target pixel position.The image A that is denoted by 702 and the image B that is denoted by703 are input to the defocus amount calculation unit 701, and thedefocus amount calculation unit 701 outputs a defocus amount 704. Thedefocus amount calculation unit 701 calculates the defocus amount bycalculating a correlation function of correlation between the image Aand the image B, or by other methods. The defocus amount is calculatedthrough the same processing that has been described above. The defocusamount 704 calculated by the defocus amount calculation unit 701 isinput to the object distance calculation unit 709. The object distancecalculation unit 709 is configured to convert the defocus amount 704into an object distance 710, and is described with reference to FIG. 9.FIG. 9 is a diagram for illustrating conversion from the image pickupplane distance to the object plane distance. Illustrated in FIG. 9 are afocused object plane 901, a lens 902, an image pickup plane 903, and animage pickup plane position 904 at which the image is defocused by adefocus amount 905. An in-focus object distance OBJ (0), an image pickup plane distance S(0) relative to a focused object, and an objectdistance OBJ (def), which is measured, are also illustrated in FIG. 9.Based on the lens equation, the following expressions are establishedrespectively for the in-focus object distance OBJ (0) and the objectdistance OBJ (def), which is measured.

$\begin{matrix}{{\frac{1}{{OBJ}\; (0)} + \frac{1}{S(0)}} = \frac{1}{f}} & (7) \\{{\frac{1}{{OBJ}({def})} + \frac{1}{{S(0)} + {def}}} = \frac{1}{f}} & (8)\end{matrix}$

Based on Expressions (7) and (8), transformation

in which OBJ(def) constitutes the left side member is performed toobtain the following expression.

$\begin{matrix}{{{OBJ}({def})} = \frac{\left( {{S(0)} + {def}} \right)*f}{\left( {{S(0)} + {def}} \right) - f}} & (9)\end{matrix}$

Expression (9) can be used in the conversion from the defocus amount tothe object distance. When the object distance 710 is output, thegradation conversion unit 711 performs data conversion suitable forwhere the object distance 710 is output to. The gradation conversionunit 711 outputs an object distance 712 on which gradation conversionhas been performed (the output is referred to as “distance map output”).The gradation conversion is described with reference to FIG. 10 on thepremise that an interface to the object distance output destination uses8-bit (0 to 255) gradation in this embodiment.

FIG. 10 is a graph for showing an example of the conversioncharacteristics of conversion from the object distance to the distancemap in the gradation conversion unit 711. In the example, the graph hasan axis of abscissa that represents the object distance and an axis ofordinate that represents the distance map, and the distance to thefocused object is 5,000 [mm]. In conversion characteristics 1001, lineargradation conversion is performed on the object distance 710. Inconversion characteristics 1002, on the other hand, non-linear gradationconversion is performed on the object distance 710 so that more bits areallocated to an object distance around 5,000 [mm], which is near anin-focus region, than in other regions. In the case where the outputdestination of the object distance is an application that requires awide distance measurement range as in Japanese Patent ApplicationLaid-Open No. 2013-243529, linear gradation conversion having theconversion characteristics 1001 or the like is performed to allocatebits equally to any object distance. In the case where the outputdestination of the object distance is an application that requiresdistance resolution near the in-focus region as in Japanese PatentApplication Laid-Open No. 2010-45457, on the other hand, non-lineargradation conversion having the conversion characteristics 1002 or thelike is performed to allocate more bits to an object distance near thein-focus region than in other regions.

According to this embodiment, it is possible to provide the imageprocessing apparatus and the image pickup apparatus capable of, when anobject distance is output based on a defocus amount, outputting anobject distance that has gradation characteristics suited to the outputdestination of the object distance. According to this embodiment, theobject distance output by the image processing apparatus and the imagepickup apparatus can therefore be used favorably where the objectdistance is output to.

Third Embodiment

A digital camera as an image pickup apparatus to which an imageprocessing apparatus according to a third embodiment of the presentinvention is applied is described below. The configuration and the imagepickup element of the digital camera as an example of the image pickupapparatus in this embodiment are the same as those described in thefirst embodiment with reference to FIGS. 1 and 2, and a descriptionthereof is omitted here unless it is necessary. The third embodimentdiffers from the first embodiment in that a plurality of defocus mapoutput destinations are provided.

FIG. 12 is a diagram for illustrating an image generation unit, adefocus amount calculation unit, and a gradation conversion unit in theimage processing unit 107 of the digital camera 100 to which the imageprocessing apparatus according to this embodiment is applied.

In FIG. 12, the image generation unit 1205 adds together a plurality ofobject images formed from luminous fluxes that arrive from differentregions of the pupil of the photographing optical system, and generatesa single object image that is generated from luminous fluxes arrivingfrom all regions of the pupil of the photographing optical system. Animage A that is denoted by 1206 and an image B that is denoted by 1207are input to the image generation unit 1205, and the image generationunit 1205 outputs an image 1208. The image 1208 generated by the imagegeneration unit 1205 receives further processing such as white balanceadjustment, color interpolation, reduction/enlargement, and filtering.The processed image is recorded on the recording medium 108 of FIG. 1 asa recorded image. The defocus amount calculation unit 1201 is configuredto calculate a defocus amount at a target pixel position. The image Athat is denoted by 1202 and the image B that is denoted by 1203 areinput to the defocus amount calculation unit 1201, and the defocusamount calculation unit 1201 outputs a defocus amount 1204. The defocusamount calculation unit 1201 calculates the defocus amount bycalculating a correlation function of correlation between the image Aand the image B, or by other methods. The defocus amount is calculatedthrough the same processing that has been described in the firstembodiment. The defocus amount 1204 calculated by the defocus amountcalculation unit 1201 is input to the gradation conversion units 1209and 1211, respectively. When the defocus amount 1204 is output, thegradation conversion units 1209 and 1211 perform data conversionsuitable to where the defocus amount 1204 is output to, by usingdifferent gradation characteristics from each other. The gradationconversion units 1209 and 1211 respectively output defocus maps 1210 and1212 on which gradation conversion has been performed. The gradationconversion is described with reference to FIGS. 5A to 5D on the premisethat an interface to each defocus map output destination uses 8-bit (0to 255) gradation in this embodiment, and that the image processing unit107 is connected to two output destinations. In the case where theoutput destination of the defocus map 1210 is an application thatrequires a wide distance measurement range as in Japanese PatentApplication Laid-Open No. 2013-243529, the gradation conversion unit1209 performs linear gradation conversion having the conversioncharacteristics 501 or the like to allocate bits equally to any defocusamount. In the case where the output destination of the defocus map 1212is an application that requires distance resolution near the in-focusregion as in Japanese Patent Application Laid-Open No. 2010-45457, onthe other hand, the gradation conversion unit 1211 performs non-lineargradation conversion having the conversion characteristics 502 or thelike to allocate more bits to a defocus amount near the in-focus regionthan in other regions.

While this embodiment gives an example in which two output units,specifically, two gradation conversion units, 1209 and 1211, areprovided, the same processing can be performed also when more than twooutput units are provided. The gradation conversion characteristics arenot limited to the conversion characteristics 501 of FIG. 5A and theconversion characteristics 502 of FIG. 5B, and may be suited to anapplication that is the output destination of the defocus amount asdescribed in the first embodiment. The output units of the thirdembodiment may output the object distance, instead of the defocusamount, as described in the second embodiment. In an alternative mode,one of the output units outputs the defocus amount and the other outputunit outputs the object distance.

According to this embodiment, it is possible to provide the imageprocessing apparatus and the image pickup apparatus capable of, when adefocus amount and an object distance are output to a plurality ofoutput destinations, outputting a defocus amount and an object distancethat each have gradation characteristics suited to the outputdestination. According to this embodiment, the defocus amount output bythe image processing apparatus and the image pickup apparatus cantherefore be used favorably where the defocus amount or the like isoutput to.

Fourth Embodiment

A digital camera as an image pickup apparatus to which an imageprocessing apparatus according to a fourth embodiment of the presentinvention is applied is described below. This embodiment gives anexample in which distance information is output to an image processingapplication, the image processing application generates a processingparameter based on the distance information on which gradationconversion has been performed, and the image processing applicationperforms processing of generating a virtual viewpoint image in which thebackground region of a photographed image is enlarged.

The digital camera as the image pickup apparatus to which the imageprocessing apparatus according to the fourth embodiment of the presentinvention is applied is described below. The configuration of thedigital camera as an example of the image pickup apparatus in thisembodiment is the same as that described in the first embodiment withreference to FIGS. 1 and 2, and a description thereof is omitted hereunless it is necessary.

The system configuration and processing of the digital camera as theimage pickup apparatus to which the image processing apparatus of thisembodiment is applied are described with reference to FIGS. 13A to 14E.As illustrated in FIG. 13A, the image processing unit 107 is providedwith an image generation unit 1305, an object distance calculation unit1301, a gradation conversion unit 1309, an image deformation parametercalculation unit 1311, and an image deformation unit 1313. Objectdistance information 1310 output from the gradation conversion unit 1309is used to divide an object into regions in the image deformation unit1313, and is used to calculate the magnification ratio of each object inthe image deformation parameter calculation unit 1311. The transmissionpath word length of the object distance information 1310 is 8 bits (0 to255). As illustrated in FIG. 13B, the image deformation unit 1313includes a region dividing unit 1321, a region-specific deformation unit1322, and an image synthesis unit 1323.

FIGS. 14A to 14E are diagrams for illustrating the processingconfiguration of the image pickup apparatus to which the imageprocessing apparatus according to this embodiment is applied. Each stepof the processing is described with reference to FIGS. 14A to 14E. FIG.14A is a diagram of a photographed image 1401. The region dividing unit1321 uses the object distance information 1310 to divide thephotographed image 1401 illustrated in FIG. 14A, and generates an image1411 (see FIG. 14B) of a region that is at the same distance as thein-focus distance (hereinafter referred to as “focused object region”),and an image 1421 (see FIG. 14C) of a plurality of background regions.Next, the regional deformation unit (deformation processing unit) 1322calculates a magnification ratio for each background object based on theobject distance information 1310, and generates a background enlargedimage 1431 (see FIG. 14D). Lastly, the image synthesis unit 1323combines the background enlarged image 1431 and the focused object image1411 to generate an image 1441.

Image deformation parameters are described with reference to FIG. 16.For a simpler description, FIG. 16 takes as an example a case where afocused object and two objects in the background are photographed.Symbols in FIG. 16 represent the following.

y₁: the size of the focused object

y₂: the size of a background object

f_(w): focal distance at the time of photographing

y_(w1), y_(w2): the size on the image plane at the time of photographing

S_(w1), S_(w2): the distance to an object at the time of photographing

f_(T): focal distance from a virtual viewpoint

Y_(T1), y_(T2): the size on the image plane of an object photographedfrom the virtual viewpoint

S_(T1), S_(T2): object distance from the virtual viewpoint

t₃: the amount of camera movement from a camera position at the time ofphotographing to a camera position at the virtual viewpoint

To generate an image in which a background object is enlarged whereasthe size of the focused object remains the same, the size Y_(T1) on theimage plane of the focused object y₁ photographed from the virtualviewpoint is equal to the size y_(w1) on the image plane of the focusedobject y₁ at the time of photographing. Therefore, based on the scalingrelation between triangles and the lens equation, Expression (10) isestablished as an expression of the magnification ratio of a background.

$\begin{matrix}{\frac{y_{T\; 2}}{y_{W\; 2}} = {\frac{f_{T}}{f_{W}}\frac{1}{{\frac{s_{W\; 1}}{s_{W\; 2}}\left( {\frac{f_{T}}{f_{W}} - 1} \right)} + 1}}} & (10)\end{matrix}$

The magnification ratio of a background is defined as N=y_(T2)/Y_(w2),and the background magnification ratio N is calculated from the focaldistance f_(w) at the time of photographing, the pieces of objectdistance information S_(w1) and S_(w2), and the virtual viewpoint focaldistance f_(T). This relation is described with reference to FIGS. 15Ato 15D. FIGS. 15A, 15C, and 15D are graphs in each of which the axis ofabscissa represents the background object distance, and the axis ofordinate represents the background magnification ratio. FIG. 15B is adiagram for illustrating an association relation between thephotographed image 1401 of FIG. 14A and a variable magnification curve1501 of FIG. 15A. The variable magnification curve 1501 of FIG. 15A is agraph that is plotted by taking the background object distance along theaxis of abscissa and the background magnification ratio along the axisof ordinate. The variable magnification curve 1501 indicates that thevalue of the background magnification ratio converges when the distancegrows to a certain level or more. Background objects that are fartherthan a prescribed distance can therefore be processed by magnificationthat uses the same magnification ratio. When the prescribed distance asa threshold for using the same magnification ratio for backgroundobjects that are located at the certain distance or more is defined asan upper limit distance S_(m), the upper limit distance S_(m) variesdepending on the focal distance at the time of photographing, thevirtual focal distance, and the object distance, and cannot bedetermined uniquely. The amount of change in magnification ratio istherefore used to determine the upper limit distance S_(m), and aconcrete description thereof is given with reference to FIG. 15C. Avariable magnification curve 1510 of FIG. 15C is the same as thevariable magnification curve 1501 of FIG. 15A. In FIG. 15C, the amountof change of the magnification ratio N with respect to a certaindistance ΔS_(w) is calculated, a distance at which the change amount issmaller than a prescribed value is searched for, and the found distanceis determined as the upper limit distance S_(m). A change amount 1513 issmaller than the prescribed value, whereas a change amount 1512 islarger than the prescribed value. A distance associated with the changeamount 1513 is accordingly determined as the upper limit distance S_(m).The prescribed value here is set in advance to a value that makes thedifference in the amount of magnification ratio change visuallynoticeable.

Gradation conversion that is performed when the object distanceinformation 1310 is output to the image deformation parametercalculation unit 1311 is described with reference to FIGS. 15A and 15D.FIG. 15D is a graph that has the object distance on the axis of abscissaand the distance map on the axis of ordinate. A variable magnificationcurve 1531 indicates gradation conversion in which the distanceresolution is higher for an object that is in proximity to the imagepickup apparatus and is accordingly large in the amount of magnificationratio change with respect to the object distance. In this gradationconversion, the distance resolution decreases gradually as the distancefrom the image pickup apparatus grows because the amount ofmagnification ratio change with respect to the object distance issmaller when the distance from the image pickup apparatus is greater. Atan object distance that exceeds the upper limit distance S_(m), adistance map value is output as the maximum value.

According to this embodiment, it is therefore possible to provide theimage processing apparatus and the image pickup apparatus capable ofoutputting distance information that has gradation characteristicsoptimum for processing in which the distance information is used.According to this embodiment, the distance information or the likeoutput by the image processing apparatus and the image pickup apparatuscan therefore be used favorably where the information is output to.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-167479, filed Aug. 20, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus, comprising: anacquiring unit configured to acquire an image signal of a plurality ofimages obtained by picking up an object image that is formed by aphotographing optical system; a determining unit configured to determinea defocus amount of the object image by using the image signal of theplurality of images; and a conversion unit configured to perform, on thedefocus amount, gradation conversion that has at least one conversioncharacteristic out of a plurality of different conversioncharacteristics, and to output information that is based on the defocusamount converted by the gradation conversion.
 2. The image processingapparatus according to claim 1, wherein the conversion unit determines agradation resolution of the output of the conversion unit based on theconversion characteristic.
 3. The image processing apparatus accordingto claim 1, wherein the plurality of different conversioncharacteristics comprise conversion characteristics with which gradationconversion varies depending on an object distance.
 4. The imageprocessing apparatus according to claim 1, wherein the plurality ofdifferent conversion characteristics comprise conversion characteristicswith which gradation conversion varies depending on an F-value of a lensof the photographing optical system.
 5. The image processing apparatusaccording to claim 1, wherein the plurality of different conversioncharacteristics comprise conversion characteristics with which gradationconversion varies depending on a base length in the image pickup of theobject image.
 6. The image processing apparatus according to claim 1,wherein the plurality of different conversion characteristics compriseconversion characteristics with which gradation conversion variesdepending on ISO sensitivity in the image pickup of the object image. 7.The image processing apparatus according to claim 1, wherein theconversion unit selects the one conversion characteristic out of theplurality of different conversion characteristics based on anapplication to which the information is output.
 8. The image processingapparatus according to claim 1, wherein the conversion unit selects theone conversion characteristic out of the plurality of differentconversion characteristics based on a photographing condition.
 9. Theimage processing apparatus according to claim 1, wherein the conversionunit comprises a generating unit configured to generate information ofan object distance from the defocus amount, and wherein the conversionunit converts the object distance based on the one conversioncharacteristic out of the plurality of different conversioncharacteristics.
 10. The image processing apparatus according to claim9, further comprising: an image dividing unit configured to acquire aplurality of divided images by dividing the image signal of theplurality of images; a deformation processing unit configured to deformeach of the plurality of divided images based on the output of theconversion unit; and an image synthesis unit configured to combine theplurality of divided images on which the deformation by the deformationprocessing unit has been performed.
 11. The image processing apparatusaccording to claim 10, wherein the conversion unit generates a distancemap value that is obtained by converting the object distance based onthe one conversion characteristic out of the plurality of differentconversion characteristics, and wherein the deformation processing unitdeforms each of the plurality of divided images based on the distancemap value output from the conversion unit.
 12. The image processingapparatus according to claim 9, wherein the conversion unit converts thedefocus amount based on each of at least two conversion characteristicsout of the plurality of different conversion characteristics, andoutputs a plurality of results obtained by the conversion of the defocusamount.
 13. The image processing apparatus according to claim 1, whereinthe image signal of the plurality of images comprise a signal selectedout of 1) an image signal of a plurality of images of the object imagedifferent from one another in blurring, 2) an image signal of aplurality of images acquired by picking up the object image fromdifferent regions of a pupil of the photographing optical system, and 3)an image signal of a plurality of images acquired by picking up imagesof an object with a plurality of image pickup units.
 14. The imageprocessing apparatus according to claim 1, wherein the defocus amount isbased on one of 1) a value that indicates a degree of match between theplurality of images and 2) a blurring evaluation value of the pluralityof images.
 15. An image pickup apparatus, comprising: an image pickupunit configured to generate an image signal by picking up an objectimage that is formed by a photographing optical system; a determiningunit configured to determine a defocus amount of the object image byusing an image signal of a plurality of images of the object image thatare generated by the image pickup unit; and a conversion unit configuredto perform, on the defocus amount, gradation conversion that has atleast one conversion characteristic out of a plurality of differentconversion characteristics, and to output information that is based onthe defocus amount converted by the gradation conversion.
 16. The imagepickup apparatus according to claim 15, further comprising: asynthesizing unit configured to combine the image signal of theplurality of images; and an image processing unit configured to processthe combined image signal based on the output of the conversion unit,wherein the image processing unit generates a processing parameter ofthe combined image signal based on the output of the conversion unit.17. The image pickup apparatus according to claim 15, wherein the imagepickup unit generates an image signal of at least a pair of images outof the object image by receiving luminous fluxes from different regionsof a pupil of the photographing optical system.
 18. The image pickupapparatus according to claim 15, wherein the image pickup unit generatesan image signal of images of the object image on different image planes.19. A method for controlling an image processing apparatus, comprising:acquiring an image signal of a plurality of images obtained by pickingup an object image that is formed by a photographing optical system;determining a defocus amount of the object image by using the imagesignal of the plurality of images; and performing, on the defocusamount, gradation conversion that has at least one conversioncharacteristic out of a plurality of different conversioncharacteristics, and outputting information that is based on the defocusamount converted by the gradation conversion.
 20. A method forcontrolling an image pickup apparatus, the image pickup apparatusincluding an image pickup unit configured to generate an image signal bypicking up an object image that is formed by a photographing opticalsystem, the method comprising: determining a defocus amount of theobject image by using an image signal of a plurality of images of theobject image that are generated by the image pickup unit; andperforming, on the defocus amount, gradation conversion that has atleast one conversion characteristic out of a plurality of differentconversion characteristics, and outputting information that is based onthe defocus amount converted by the gradation conversion.
 21. Acomputer-readable storage medium having recorded thereon a program forcausing a computer so that the computer executes a method forcontrolling an image processing apparatus, the method comprising:acquiring an image signal of a plurality of images obtained by pickingup an object image that is formed by a photographing optical system;determining a defocus amount of the object image by using the imagesignal of the plurality of images; and performing, on the defocusamount, gradation conversion that has at least one conversioncharacteristic out of a plurality of different conversioncharacteristics, and outputting information that is based on the defocusamount converted by the gradation conversion.
 22. A computer-readablestorage medium having recorded thereon a program for causing a computerso that the computer executes a method for controlling an image pickupapparatus, the image pickup apparatus including an image pickup unitconfigured to generate an image signal by picking up an object imagethat is formed by a photographing optical system, the method comprising:determining a defocus amount of the object image by using an imagesignal of a plurality of images of the object image that are generatedby the image pickup unit; and performing, on the defocus amount,gradation conversion that has at least one conversion characteristic outof a plurality of different conversion characteristics, and outputtinginformation that is based on the defocus amount converted by thegradation conversion.